• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    Embodiments of the present provide a method and a device for processing an image, server and storage medium. The method includes: determining (110), based on an object type of an object in an image to be processed, a feature expression of the object in the image to be processed; and determining (120) an entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed and a feature expression of an entity in a knowledge graph.
    公开号:EP3709212A1
    申请号:EP20150891.8
    申请日:2020-01-09
    公开日:2020-09-16
    发明作者:Xiaohan ZHANG；Ye XU；Kexin REN；Zhifan Feng；Yang Zhang；Yong Zhu
    申请人:Beijing Baidu Netcom Science and Technology Co Ltd；
    IPC主号:G06K9-00
    专利说明:
    [0001] The present disclosure relates to the field of image processing, and more particularly, to a method and a device for processing an image, a server and a storage medium. BACKGROUND
    [0002] With rapid increasing of image and video data on the Internet, the demand for image processing technology increases. The image processing technology may be used to quickly understand content of pictures or videos, to labeling pictures or videos with corresponding tags, and to recommend certain kinds of pictures or videos to users.
    [0003] Currently, a method of labelling an image based on an image processing technology to obtain a result of labelling is mainly to set an object tag and to analyze related actions. However, this method may only analyze the content shallowly, and cannot analyze the content deeply to obtain a deep result. As a result, the result of labelling may be non-semantic, which leads to an inaccurate analysis result. In addition, a method of labelling an image using external knowledge graph data to obtain the result of labelling improves the identification effect of tags. However, the tag may be non-semantic, and the picture content cannot be deeply understood. SUMMARY
    [0004] Embodiments of the present disclosure provide a method and a device for processing an image, a server and a storage medium, to deeply understand the image using a knowledge graph and to obtain a structured semantic tag of the image.
    [0005] Embodiments of the present disclosure provide a method for processing an image. The method includes: determining, based on an object type of an object in an image to be processed, a feature expression of the object in image to be processed; and determining an entity associated with the object in the image to be processed based on the feature expression of object in the image to be processed and a feature expression of an entity in a knowledge graph.
    [0006] Embodiments of the present disclosure provide a device for processing an image. The device includes: a feature expression determination module, configured to determine, based on an object type of an object in an image to be processed, a feature expression of the object in the image to be processed; and an associated entity determination module, configured to determine an entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed and a feature expression of an entity in a knowledge graph.
    [0007] Embodiments of the present disclosure provide a computer readable storage medium having computer programs stored thereon. When the programs are executed by a processor, the method for processing an image according to above embodiments of the present disclosure is implemented.
    [0008] With embodiments of the present disclosure, based on the object type of the object in the image to be processed, the feature expression of the object in the image to be processed is determined, the entity associated with the object in the image to be processed is determined based on the feature expression of object in the image to be processed and the feature expression of the entity in the knowledge graph. Determining the entity associated with the object in the image to be processed based on the feature expression of the object and the feature expression of the entity in the knowledge graph enables deep understanding of the image, thereby facilitating expansion and verification of the entity associated with the object and realizing determination of a structured semantic tag. BRIEF DESCRIPTION OF THE DRAWINGS
    [0009] FIG. 1 is a flowchart illustrating a method for processing an image according to embodiments of the present disclosure. FIG. 2 is a flowchart illustrating a method for processing an image according to embodiments of the present disclosure. FIG. 3 is a schematic diagram of determining an entity associated to an image according to embodiments of the present disclosure. FIG. 4 is a schematic diagram illustrating a device for processing an image according to embodiments of the present disclosure. FIG. 5 is a schematic diagram illustrating a server according to embodiments of the present disclosure.DETAILED DESCRIPTION
    [0010] The present disclosure will be described in detail below with reference to the accompanying drawings and the embodiments. It should be understood that, the specific embodiments described herein are only used to explain the present disclosure rather than to limit the present disclosure. In addition, it should also be noted that, for convenience of description, only part but not all structures related to the present disclosure are illustrated in the accompanying drawings.
    [0011] FIG. 1 is a flowchart illustrating a method for processing an image according to embodiments of the present disclosure. The method according to embodiments may be applicable for deeply understanding an image to further determine an image tag. The method may be implemented by a device for processing an image. The device may be implemented by software and/or hardware, and the device may be integrated in a server. As illustrated in FIG. 1, the method according to embodiments may include the following.
    [0012] At block 110, a feature expression of an object in an image to be processed is determined based on an object type of the object in the image to be processed.
    [0013] In detail, the image to be processed may be a picture or an image from a video. Features of the object may include entities in the image to be processed. In an example, determining, based on the object type of the object in the image to be processed, the feature expression of the object in the image to be processed may include determining the feature expression of the object in the image to be processed based on the object type and an object image area of the object included in the image to be processed. Exemplarily, target identification may be performed on the image to be processed and multiple objects in the image to be processed may be identified, classified and delineated, to obtain the object type and the object image area of each object. Pixel data of the image to be processed may be input into an identification model to obtain the object type of each object. Further, the pixel data may be input into a depth learning model corresponding to each object type to perform feature extraction on the image to be processed and to obtain the feature expression. In an example, the pixel data may be input into a convolutional neural network (CNN) model for feature extraction to obtain the feature expression corresponding to each object. In some examples, the object of each object type is input to a deep learning model corresponding to the object type to extract features to obtain the feature expression of the object. For example, an image of a human face type is input into a face identification model, such as DeepID, DeepFace or FaceNet to perform feature extraction on the human face to obtain the feature expression of the human face.
    [0014] At block 120, an entity associated with the object in the image to be processed is determined based on the feature expression of the object in the image to be processed and the feature expression of the entity in the knowledge graph.
    [0015] In detail, the feature expression of the object may be object features displayed in the image to be processed, such as shape, color, and texture of the object. The feature expression of the object in the image to be processed is matched with feature expressions in the knowledge graph, and an entity in the knowledge graph matching with the feature expression of the object in the image to be processed may be determined as the entity associated with the object in the image to be processed. In an example, determining the entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed and the feature expression of the entity in the knowledge graph may include: determining the entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed, a feature expression of the image to be processed, and a feature expression of text associated with the image to be processed, the feature expression of the entity in the knowledge graph and entity attribute information. Exemplarily, in practical applications, an image provided on the Internet usually has text associated with the images, for example, text information associated with the image to be processed, such as text content, a source title, and an article title corresponding to the image. The feature expression of the text is obtained. The entity is further determined by combining the feature expression of the text. The entity attribute information of the entity in the knowledge graph includes an essential attribute of the object, for example, attribute information of a person in the image to be processed such as age, gender, and occupation. The feature expression of the object in the image to be processed, the feature expression of the image to be processed, the feature expression of the text associated with the image to be processed are matched with the feature expression of the entity in the knowledge graph and the entity attribute information and feature expressions are compare, to determine the entity associated with the object in the image to be processed.
    [0016] In an example, after the entity associated with the object in the image to be processed is determined based on the feature expression of the object in the image to be processed and the feature expression of the entity in the knowledge graph, the method further includes: determining a feature expression of an article based on an entity in the article to which the image to be processed belongs and determining a relevance between the article and the image based on the feature expression of the article and the entity associated with the object in the image to be processed. In detail, it is possible that a low relevance exists between the article to which the image to be processed belongs and the image to be processed. It is likely that the image and the article are linked due to identification errors of the image or the article. Therefore, it may be determined whether the image to be processed is relevant to the article based on the relevance. Exemplarily, the feature expression of the article in the image to be processed is determined, and the relevance between the article and the image to be processed is further determined based on the feature expression of the article and the entity associated with the object in the image to be processed.
    [0017] With the technical solution according to embodiments of the present disclosure, the feature expression of the object in an image to be processed is determined based on the object type of the object in the image to be processed, and the entity associated with the object in the image to be processed is determined based on the feature expression of the object in the image to be processed and the feature expression of the entity in the knowledge graph. Determining the entity associated with the object in the image to be processed based on the feature expression of the object and the feature expression of the entity in the knowledge graph enables deep understanding of the image, thereby facilitating expansion and verification of the entity associated with the object and realizing determination of a structured semantic tag.
    [0018] FIG. 2 is a flowchart illustrating a method for processing an image according to embodiments of the present disclosure.
    [0019] This embodiment is optimized on the basis of embodiment 1, and details which are not described in this embodiment can be referred to the above embodiment. As illustrated in FIG. 2, the method according to this embodiment may include the following.
    [0020] At block 210, a feature expression of an object in an image to be processed is determined based on an object type of the object in the image to be processed.
    [0021] At block 220, an entity associated with the object in the image to be processed is determined based on the feature expression of the object in the image to be processed and a feature expression of an entity in a knowledge graph.
    [0022] At block 230, a first determination manner is determined as a determination manner of the entity determined.
    [0023] In detail, the first determination manner may be the manner in the block 220 for determining the entity. That is, the entity associated with the object in the image to be processed is determined based on the feature expression of the object in the image to be processed and the feature expression of the entity in the knowledge graph.
    [0024] At block 240, the entity associated with the object in the image to be processed is determined by at least one secondary determination manner.
    [0025] In detail, the secondary determination manner may be a determination manner other than the determination manner of the block 220. For example, an image provided on the Internet usually has text associated with the image, such as, text information like text content, a source title, and an article title corresponding to the image. Text entities can be obtained through a feature expression of the text. However, a relevance between the text and the image may be low. Therefore, a manner of determining the associated entity based on the text information relevant to the image to be processed, with a secondary method of determining an image relevance result based on the text information, may be determined as a secondary determination manner.
    [0026] In an example, determining the entity associated with the object in the image to be processed based on at least one secondary determination manner may include matching the image to be processed with an image of a candidate entity to determine the entity associated with the image to be processed. In an example, determining the entity associated with the object in the image to be processed based on at least one secondary determination manner may include matching the text to which the image to be processed belongs with the knowledge graph to determine the entity associated with the image to be processed. Exemplarily, the image to be processed is matched with the image of the candidate entity. When a matching result satisfies conditions, it may be determined that the candidate entity is associated with the image to be processed. In some examples, the text of the article to which the image to be processed belongs is matched with the knowledge graph. When a matching result satisfies conditions, it may be determined that a matched entity is the entity associated with the image to be processed.
    [0027] For example, the image to be processed is matched with the image of the candidate entity to determine an actor entity in the image to be processed. In another example, the text of the article to which the image belongs is matched with the knowledge graph to determine the actor entity associated with the text entity in the article.
    [0028] At block 250, the entity associated with the object in the image to be processed is re-determined based on the determination manner and determination frequency of each entity.
    [0029] For example, the determination manner and the determination frequency corresponding to the determination manner may be added to the feature expression of the entity for determining the entity associated with the object in the image to be processed, such that the entity associated with the object in the image to be processed is re-determined. For example, based on the re-determined actor entity, a character entity is determined from the knowledge graph. The character entity may be added to the entity associated with the object in the image to be processed.
    [0030] At block 260, new entities having edge relations with the entity associated with the object are selected from the knowledge graph.
    [0031] In detail, the entity associated with the object is verified based on the new entities having the edge relations with the entity associated with the object in the knowledge graph. For example, an entity in the knowledge graph is a TV series named "To the Sky Kingdom", the entity having the edge relation may be: Bai Qian (a character). That is, the character of Bai Qian has the edge relation with the TV series named "To the Sky Kingdom". By selecting the new entities having the edge relations with the entity associated with the object, attributes of the entity may be deeply understood.
    [0032] At block 270, an updated entity associated with the image is selected from the new entities based on a relation among the new entities.
    [0033] In detail, the relation among the new entities is obtained. For example, an intersection operation is performed on the new entities. The intersection of the new entities may be used as the updated entity associated with the image. For example, if three entities are associated with one image, three sets of new entities having the edge relation with the three entities respectively are selected from the knowledge graph. The intersection of the three sets of new entities corresponding t the three entities are used as the updated entity associated with the image. Exemplarily, the TV series that each actor entity in the image has starred in are obtained. The intersection of the TV series are obtained as the updated entity associated with the image.
    [0034] In an example, FIG. 3 is a schematic diagram of determining an entity associated with an image according to Embodiment 2 of the present disclosure. As illustrated in FIG. 3, an entity associated with an object in an image to be processed is determined. An entity corresponding to the object in the image is obtained by image identification and classification, and feature extraction. For example, entities in the image are identified as actors, namely Liu Bei, Guan Yu, and Zhang Fei. Based on entities in the knowledge graph that have the edge relation with Liu Bei, Guan Yu, and Zhang Fei, it may be determined that each character corresponds to the TV series named "Romance of the Three Kingdoms". Further, it may be determined that plot corresponding to the image is Peach Garden Oath, based on the TV series and the three characters. In an example, the entities corresponding to the object in the image are obtained by image identification and classification, and feature extraction. For example, entities in the image are identified as actors, namely Liu Bei, Guan Yu, and Zhang Fei. A worship action is identified from the image to be processed. It may be determined that the plot corresponding to the image to be processed is Peach Garden Oath. It should be noted that in FIG. 3, in order to distinguish the objects displayed from the background, a white rectangle is used to indicate the object image area, which is not specifically limited.
    [0035] In embodiments of the present disclosure, the first determination manner is determined as the determination mode of determining the entity; the entity associated with the object in the image to be processed is determined by the at least one secondary determination manner; and the entity associated with the object in the image to be processed is re-determined based on the determination manner and determination frequency of each entity. The new entities having the edge relation with the entity associated with the object are selected from the knowledge graph. The updated entity associated with the image is selected from the new entities based on the relations among the new entities. Through the secondary determination manner, the image may be deeply understood and the entity associated with the image may be expanded. The updated entity associated with the image is determined based on the relation among the new entities, thereby implementing deep analysis and verification of the image and accurately obtaining a semantic tag of the image.
    [0036] FIG. 4 is a schematic diagram illustrating a device for processing an image according to embodiments of the present disclosure. The device is applicable for deeply understanding images to further determine an image tag. The device may be implemented by software and/or hardware, and may be integrated in a server. As illustrated in FIG. 4, the device may further include: a feature expression determination module 310, and an associated entity determination module 320.
    [0037] The feature expression determination module 310 may be configured to determine, based on an object type of an object in an image to be processed, a feature expression of the object in the image to be processed.
    [0038] The associated entity determination module 320 may be configured to determine an entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed and a feature expression of an entity in a knowledge graph.
    [0039] In an example, the feature expression determination module 310 may be configured to determine the feature expression of the object in the image to be processed, based on the object type of the object in the image to be processed and an object image area.
    [0040] In an example, the associated entity determination module 320 may be configured to determine the entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed, a feature expression of the image to be processed, a feature expression of text associated with the image to be processed, the feature expression of the entity in the knowledge graph and entity attribute information.
    [0041] In an example, the device may further include: an article feature expression determination module, and a relevance determination module.
    [0042] The article feature expression determination module may be configured to determine a feature expression of an article based on an entity in the article to which the image to be processed belongs.
    [0043] The relevance determination module may be configured to determine a relevance between the article and the image based on the feature expression of the article and the entity associated with the object in the image to be processed.
    [0044] In an example, the device may further include: a determination manner determining module, a secondary determination module, and an associated entity redetermination module.
    [0045] The determination manner determining module may be configured to determine a first determination manner as a determination manner of determining the entity.
    [0046] The secondary determination module may be configured to determine the entity associated with the object in the image to be processed by at least one secondary determination manner.
    [0047] The associated entity redetermining module may be configured to re-determine the entity associated with the object in the image to be processed based on the determination manner and determination frequency of each entity.
    [0048] In an example, the secondary determination module may be further configured to match the image to be processed with an image of a candidate entity to determine the entity associated with the image to be processed; and/or, match a text to which the image to be processed belongs with the knowledge graph to determine the entity associated with the image to be processed.
    [0049] In an example, the device may further include a selecting module and an updated entity selecting module.
    [0050] The selecting module is configured to select new entities having edge relations with the entity associated with the object from the knowledge graph.
    [0051] The updated entity selecting module may be configured to select an updated entity associated with the image from the new entities based on a relation among the new entities.
    [0052] The device for processing an image according to above embodiments is used to perform the method for processing an image according to any of the above embodiments, and the technical principle and the generated technical effect are similar, which are not described herein again.
    [0053] FIG. 5 is a schematic diagram illustrating a server according to embodiments of the present disclosure. FIG. 5 illustrates a block diagram of an exemplary server 412 applicable for implementing embodiments of the present disclosure. The server 412 illustrated in FIG. 5 is merely an example and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.
    [0054] As illustrated in Fig. 5, the computer device 412 may be represented via a general computer device form. Components of the computer device 412 may include but be not limited to one or more processors or processing units 416, a system memory 428, a bus 418 connecting various system components including the system memory 428 and the processing units 416.
    [0055] The bus 418 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, an Industry Standard Architecture (hereinafter referred to as ISA) bus, a Micro Channel Architecture (hereinafter referred to as MAC) bus, an enhanced ISA bus, a Video Electronics Standards Association (hereinafter referred to as VESA) local bus and Peripheral Component Interconnection (PCI) bus.
    [0056] The computer device 412 typically includes a variety of computer system readable media. These media may be any available media accessible by the computer device 412 and includes both volatile and non-volatile media, removable and non-removable media.
    [0057] The system memory 428 may include a computer system readable medium in the form of volatile memory, such as a random access memory (hereinafter referred to as RAM) 430 and/or a high speed cache memory 432. The computer device 412 may further include other removable or non-removable, volatile or non-volatile computer system storage media. By way of example only, the storage system 434 may be configured to read and write a non-removable and non-volatile magnetic media (not shown in FIG. 5, commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk driver for reading from and writing to a removable and non-volatile magnetic disk (such as "floppy disk") and a disk driver for a removable and non-volatile optical disk (such as compact disk read only memory (hereinafter referred to as CD-ROM), Digital Video Disc Read Only Memory (hereinafter referred to as DVD-ROM) or other optical media) may be provided. In these cases, each driver may be connected to the bus 418 via one or more data medium interfaces. The memory 428 may include at least one program product. The program product has a set (such as, at least one) of program modules configured to perform the functions of various embodiments of the present disclosure.
    [0058] A program/utility 440 having a set (at least one) of the program modules 442 may be stored in, for example, the memory 428. The program modules 442 include but are not limited to, an operating system, one or more application programs, other programs modules, and program data. Each of these examples, or some combination thereof, may include an implementation of a network environment. The program modules 442 generally perform the functions and/or methods in the embodiments described herein.
    [0059] The computer device 412 may also communicate with one or more external devices 414 (such as, a keyboard, a pointing device, a display 424, etc.). Furthermore, the computer device 412 may also communicate with one or more communication devices enabling a user to interact with the computer device 412 and/or other devices (such as a network card, modem, etc.) enabling the computer device 412 to communicate with one or more computer devices. This communication can be performed via the input/output (I/O) interface 422. Also, the computer device 412 may communicate with one or more networks (such as a local area network (hereafter referred to as LAN), a wide area network (hereafter referred to as WAN) and/or a public network such as an Internet) through a network adapter 420. As shown, the network adapter 420 communicates with other modules of the computer device 412 over the bus 418. It should be understood that, although not shown, other hardware and/or software modules may be used in connection with the computer device 412. The hardware and/or software includes, but is not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tap Drive and data backup storage system.
    [0060] The processing unit 416 is configured to execute various functional applications and data processing by running programs stored in the system memory 428, for example, implementing the method for processing an image according to embodiments of the present disclosure.
    [0061] Embodiments of the present disclosure provide a computer readable storage medium having computer programs stored thereon. When the programs are executed by a processor, the method for processing an image may be implemented.
    [0062] The computer storage medium according to the embodiments of the present disclosure may adopt any combination of one or more computer-readable medium. A computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, but is not limited to, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, component or any combination thereof. A specific example of the computer readable storage media include (a non-exhaustive list): an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an Erasable Programmable Read Only Memory (EPROM) or a flash memory, an optical fiber, a compact disc read-only memory (CD-ROM), an optical memory component, a magnetic memory component, or any suitable combination thereof. In context, the computer readable storage medium may be any tangible medium including or storing programs. The programs may be used by an instruction executed system, apparatus or device, or a combination thereof.
    [0063] The computer readable signal medium may include a data signal propagating in baseband or as part of a carrier which carries computer readable program codes. Such propagated data signal may be in many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer readable signal medium may also be any computer readable medium other than the computer readable storage medium, which may send, propagate, or transport programs used by an instruction executed system, apparatus or device, or a connection thereof.
    [0064] The program code stored on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.
    [0065] The computer program code for carrying out operations of embodiments of the present disclosure may be written in one or more programming languages. The programming language includes an object oriented programming language, such as Java, Smalltalk, C ++, as well as conventional procedural programming language, such as "C" language or similar programming language. The program code may be executed entirely on a user's computer, partly on the user's computer, as a separate software package, partly on the user's computer, partly on a remote computer, or entirely on the remote computer or server. In a case of the remote computer, the remote computer may be connected to the user's computer or an external computer (such as using an Internet service provider to connect over the Internet) through any kind of network, including a Local Area Network (hereafter referred as to LAN) or a Wide Area Network (hereafter referred as to WAN).
    [0066] It should be noted that, the above are only preferred embodiments and applied technical principles of the present disclosure. Those skilled in the art should understand that, the present disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that are made by those skilled in the art will not depart from the scope of the present disclosure. Therefore, although the present disclosure has been described in detail by the above embodiments, the present disclosure is not limited to the above embodiments, and more other equivalent embodiments may be included without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.
    权利要求:
    Claims (15)
    [0001] A method for processing an image, comprising:
    determining (110; 210), based on an object type of an object in an image to be processed, a feature expression of the object in the image to be processed; and
    determining (120; 220) an entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed and a feature expression of an entity in a knowledge graph.
    [0002] The method of claim 1, wherein determining (110; 210), based on the object type of the object in the image to be processed, the feature expression of the object in the image to be processed comprises:determining the feature expression of the object in the image to be processed, based on the object type of the object in the image to be processed and an object image area.
    [0003] The method of claim 1 or 2, wherein determining (120; 220) the entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed and the feature expression of the entity in the knowledge graph comprises:determining the entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed, a feature expression of the image to be processed, and a feature expression of text associated with the image to be processed, the feature expression of the entity in the knowledge graph and entity attribute information.
    [0004] The method of any one of claims 1 to 3, further comprising:
    determining a feature expression of an article based on an entity in the article to which the image to be processed belongs; and
    determining a relevance between the article and the image based on the feature expression of the article and the entity associated with the object in the image to be processed.
    [0005] The method of any one of claims 1-4, further comprising:
    determining (230) a first determination manner as a determination manner of determining the entity;
    determining (240) the entity associated with the object in the image to be processed by at least one secondary determination manner; and
    re-determining (250) the entity associated with the object in the image to be processed based on a determination manner and determination frequency of each entity;
    the determination manner comprising a first determination manner and at least one secondary determination manner.
    [0006] The method of claim 5, wherein determining (240) the entity associated with the object in the image to be processed by the at least one secondary determination manner comprises:
    matching the image to be processed with an image of a candidate entity to determine the entity associated with the image to be processed; and/or,
    matching text to which the image to be processed belongs with the knowledge graph to determine the entity associated with the image to be processed.
    [0007] The method of any one of claims 1 to 6, further comprising:
    selecting (260) new entities having edge relations with the entity associated with the object from the knowledge graph; and
    selecting (270) an updated entity associated with the image from the new entities based on a relation among the new entities.
    [0008] A device for processing an image, comprising:
    a feature expression determination module (310), configured to determine, based on an object type of an object in an image to be processed, a feature expression of the object in the image to be processed; and
    an associated entity determination module (320), configured to determine the entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed and a feature expression of an entity in a knowledge graph.
    [0009] The device of claim 8, wherein the feature expression determination module (310) is further configured to:determine the feature expression of the object in the image to be processed, based on the object type of the object in the image to be processed and an object image area.
    [0010] The device of claim 8 or 9, wherein the associated entity determination module (320) is further configured to:determine the entity associated with the object in the image to be processed based on the feature expression of the object in the image to be processed, a feature expression of the image to be processed, and a feature expression of text associated with the image to be processed, the feature expression of the entity in the knowledge graph and entity attribute information.
    [0011] The device of any one of claims 8 to 10, further comprising:
    an article feature expression determination module, configured to determine a feature expression of an article based on an entity in the article to which the image to be processed belongs; and
    a relevance determination module, configured to determine a relevance between the article and the image based on the feature expression of the article and the entity associated with the object in the image to be processed.
    [0012] The device of any one of claims 8 to 11, further comprising:
    a determination manner determining module, configured to determine a first determination manner as a determination manner of determining the entity;
    a secondary determination module, configured to determine the entity associated with the object in the image to be processed by at least one secondary determination manner; and
    an associated entity redetermination module, configured to re-determine the entity associated with the object in the image to be processed based on a determination manner and determination frequency of each entity;
    the determination manner comprising a first determination manner and at least one secondary determination manner.
    [0013] The device of claim 12, wherein the secondary determination module is further configured to:
    match the image to be processed with an image of a candidate entity to determine the entity associated with the image to be processed; and/or,
    match text to which the image to be processed belongs with the knowledge graph to determine the entity associated with the image to be processed.
    [0014] The device of any one of claims 8 to 13, further comprising:
    a selecting module, configured to select new entities having edge relations with the entity associated with the object from the knowledge graph; and
    an updated entity selecting module, configured to select an updated entity associated with the image from the new entities based on a relation among the new entities.
    [0015] A computer readable storage medium having computer programs stored thereon, wherein when the programs are executed by a processor, the method for processing an image of any one of claims 1-7 is implemented.
    类似技术:
    公开号 | 公开日 | 专利标题
    RU2688271C2|2019-05-21|Image search in natural language
    AU2016203856B2|2017-02-23|System and method for automating information abstraction process for documents
    US20170357720A1|2017-12-14|Joint heterogeneous language-vision embeddings for video tagging and search
    CN108304835B|2019-12-06|character detection method and device
    US10726212B2|2020-07-28|Presenting translations of text depicted in images
    US9971763B2|2018-05-15|Named entity recognition
    US10055403B2|2018-08-21|Rule-based dialog state tracking
    US20180173692A1|2018-06-21|Iconographic symbol predictions for a conversation
    US9519641B2|2016-12-13|Photography recognition translation
    US9256795B1|2016-02-09|Text entity recognition
    US9471874B2|2016-10-18|Mining forums for solutions to questions and scoring candidate answers
    US9082035B2|2015-07-14|Camera OCR with context information
    US9239833B2|2016-01-19|Presenting translations of text depicted in images
    WO2018028583A1|2018-02-15|Subtitle extraction method and device, and storage medium
    KR101160597B1|2012-06-28|Content retrieval based on semantic association
    US8577882B2|2013-11-05|Method and system for searching multilingual documents
    JP4461769B2|2010-05-12|Document retrieval / browsing technique and document retrieval / browsing device
    US9633479B2|2017-04-25|Time constrained augmented reality
    US9619735B1|2017-04-11|Pure convolutional neural network localization
    US20160005171A1|2016-01-07|Image Analysis Device, Image Analysis System, and Image Analysis Method
    US20070255706A1|2007-11-01|Information retrieval apparatus
    US7668371B2|2010-02-23|System and method for adaptively separating foreground from arbitrary background in presentations
    CN103079092B|2015-12-23|Obtain the method and apparatus of people information in video
    US10573039B2|2020-02-25|Techniques for incorporating a text-containing image into a digital image
    KR20150121483A|2015-10-29|Content processing method and electronic device thereof
    同族专利:
    公开号 | 公开日
    KR20200109239A|2020-09-22|
    JP2020149686A|2020-09-17|
    US20200294267A1|2020-09-17|
    CN109918513A|2019-06-21|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    法律状态:
    2020-08-14| STAA| Information on the status of an ep patent application or granted ep patent|Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
    2020-08-14| PUAI| Public reference made under article 153(3) epc to a published international application that has entered the european phase|Free format text: ORIGINAL CODE: 0009012 |
    2020-09-16| AK| Designated contracting states|Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
    2020-09-16| AX| Request for extension of the european patent|Extension state: BA ME |
    2021-03-19| STAA| Information on the status of an ep patent application or granted ep patent|Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
    2021-04-21| RBV| Designated contracting states (corrected)|Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
    2021-04-21| 17P| Request for examination filed|Effective date: 20210315 |
    优先权:
    申请号 | 申请日 | 专利标题
    [返回顶部]